Improving Deep Neural Network with Multiple Parametric Exponential Linear Units

Activation function is crucial to the recent successes of neural network. In this paper, we propose a new activation function that generalizes and unifies the rectified and exponential linear units. The proposed method, named MPELU, has the advantages of PReLU and ELU. We show that by introducing learnable parameters like PReLU, MPELU provides better generalization capability than PReLU and ELU. As a generalization of ELU, MPELU maintains the ability of reducing bias shift and shows better convergence than ReLU and PReLU. In addition, we put forward a way of initialization suitable for exponential linear units. To the best of our knowledge, the proposed initialization is the first one for exponential-linear-unit networks. Experiments prove that our initialization is helpful to train very deep exponential-linear-unit networks. And exponential-linear-unit networks with the proposed initialization, to some extent, are able to deal with the degradation phenomenon caused by depth.